Skip to content

[DSpark] Support pipeline-parallel targets in aggregated serving - #56956

Merged
vllm-bot merged 3 commits into
vllm-project:mainfrom
lucifer1004:pr/dspark-pp-ifb
Sep 23, 2026
Merged

[DSpark] Support pipeline-parallel targets in aggregated serving#56956
vllm-bot merged 3 commits into
vllm-project:mainfrom
lucifer1004:pr/dspark-pp-ifb

Conversation

@lucifer1004

@lucifer1004 lucifer1004 commented Sep 15, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

What

DSpark on a pipeline-parallel target in aggregated (non-PD) serving. The drafter runs wholly on the last PP stage; this PR makes that work correctly:

  • Draft broadcast across stages: the last stage's speculator proposes drafts, but earlier stages need them for the next verification step. The last rank now broadcasts the fresh drafts on the PP group (PPHandler.broadcast_drafts), and the broadcast is gated out of the no-op second call site when a speculator ran (double-posting misaligned the recv FIFO and hung the pipeline).
  • Warmup: the deferred PP post-update triton kernel never runs during warmup steps on non-last ranks; its first mid-serving compile would deadlock against the in-flight NCCL broadcast. warmup_pp_decode_update compiles it during capture, and the sampled-token broadcast is disabled for the warmup window.
  • Draft embedding under PP: the drafter aliases the target's embedding table, which [Core][MRV2] Support eagle3 spec decode with pipeline parallel #50514 retains on the last PP stage when speculative decoding is active. (An earlier revision of this PR loaded a separate drafter copy; dropped per review.)
  • Padded graph batch safety: the DFlash prepare-inputs kernel clears input_ids/positions and sets is_padding on CUDA-graph padding rows, and the DSv4 top-k router zeroes padded-row selections instead of reading uninitialized state.

Not a duplicate

Split from #53577 (reviewer request): this PR is the aggregated-serving half; the PD prefill + KV-transfer half follows separately.

Tests

  • New/extended: tests/v1/worker/test_pp_utils.py, tests/v1/spec_decode/test_dflash_prepare_inputs.py, tests/kernels/moe/test_topk_softplus_sqrt.py.
  • pytest tests/v1/worker/test_pp_utils.py tests/v1/spec_decode/test_dflash_prepare_inputs.py tests/kernels/moe/test_topk_softplus_sqrt.py -q → 1686 passed, 6 skipped.
  • pytest tests/v1/worker/test_gpu_warmup_blocks.py tests/v1/worker/test_gpu_model_runner_v2.py -q → 34 passed.

Model evaluation

DeepSeek-V4-Flash (0731) DSpark K=5, IFB PP2×TP2, GSM8K strict-match: 0.9484 (1319 samples; an earlier revision measured 0.9545 — both within noise of the target-only arm). Spec decode mean acceptance length 4.38.

Kimi-K3 DSpark IFB PP relies on the TritonMLA causal multi-token fix from #51065 (merged; included in this branch's base). Smoke-verified on SM120 (PP2×TP4): coherent outputs with healthy per-position draft acceptance (144/96/55/31/22/17/12/6 over K=8).

AI assistance was used in preparing this PR.

claude[bot]
claude Bot reviewed Sep 15, 2026
View reviewed changes

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

@mergify mergify Bot added deepseek Related to DeepSeek models kimi k3 DSv4 speculative-decoding dflash mrv2 Model Runner V2 specific labels Sep 15, 2026
This was referenced Sep 15, 2026
Open
Closed
GirasoleY
GirasoleY reviewed Sep 15, 2026
View reviewed changes
Comment thread vllm/models/kimi_k3/nvidia/dspark_mla.py Outdated
Comment on lines +163 to +168
if _target_pp_world_size() > 1:
self.embed_tokens = VocabParallelEmbedding(
self.config.vocab_size,
self.config.hidden_size,
prefix=maybe_prefix(prefix, "embed_tokens"),
)

@GirasoleY GirasoleY Sep 15, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#50514 added token embedding at pp last stage, I think we don't need this anymore (same for the dsv4 path)

@lucifer1004 lucifer1004 Sep 15, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed — removed in c6a9343. loads_own_embed_under_pp and the load_own_embed block are gone from both the K3 (dspark_mla.py) and DSv4 (dspark.py) drafters; the draft now relies on #50514's token embedding provided at the last PP stage.

Verification on this branch:

  • Unit: the 60 dspark/PP-related tests in tests/config/test_dspark_prefill_only.py, tests/v1/worker/test_gpu_model_runner_v2.py and tests/v1/worker/test_gpu_warmup_blocks.py all pass.
  • e2e (IFB, aggregated serving): DeepSeek-V4-Flash DSpark pp2tp2 GSM8K = 0.9484 (strict-match, 1319 samples; vs 0.9545 measured on this branch before the removal — within noise), spec decode mean acceptance length 4.38.
  • Kimi-K3 DSpark pp2tp4 smoke (with [Bugfix][MLA] TritonMLA: fix illegal memory access on causal multi-token decode #51065): coherent outputs across math/Chinese/code prompts and healthy per-position acceptance (144/96/55/31/22/17/12/6). Note: with [Core][MRV2] Support eagle3 spec decode with pipeline parallel #50514 the embedding is shared from the target, so the drafter also saves one embedding copy of memory versus the old self-loading path.

Comment thread vllm/v1/worker/gpu_worker.py
# window and restore it before serving.
pp_handler = getattr(self.model_runner, "pp_handler", None)
if pp_handler is not None:
pp_handler.set_disabled(True)

@GirasoleY GirasoleY Sep 15, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you help me understand how the deadlock happen? If it's about collective running on multiple streams, I'd prefer enforce collective ordering instead of skip warm up here.

@lucifer1004 lucifer1004 Sep 15, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Happy to walk through it. The deadlock chain we observed (5-11 min hangs, reproducible):

  1. The sampled-token broadcast is posted on a side stream: non-last ranks post NCCL recv kernels that spin on-device until the last rank's broadcast lands (early posting is by design for pipelining).
  2. Warmup runs synthetic steps whose sampled outputs are discarded — those steps post the recv spins but carry no matching payload, so the spins linger.
  3. Warmup is also where most first-time Triton JIT loads happen. Even on a cubin cache hit, a new specialization key (alignment/divisibility) lazily triggers a fresh cuModuleLoadData — and that call blocks at context level behind the spinning NCCL kernel (driver behavior we can't change from user space).
  4. The blocked host thread never advances to the next step, so it never sends its activation p2p; the peer stage waits on those activations; the spinning recv never completes. Cycle → pipeline hang.

On enforcing collective ordering instead: ordering the broadcast against the compute stream would reintroduce a per-step serialization cost in serving to fix a warmup-only hazard, and it still wouldn't cover the module-load half of the cycle (a new shape specialization mid-serving would wedge behind a legitimately in-flight broadcast — that's why we precompile the deferred post-update kernel during capture via warmup_pp_decode_update, with alignment-matched views). The disable window here is scoped to synthetic warmup steps that carry no payload by construction, so serving behavior is untouched. If you still see a cleaner ordering-based formulation I'm happy to try it.

@mergify

mergify Bot commented Sep 23, 2026

Copy link
Copy Markdown
Contributor

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @lucifer1004.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@mergify mergify Bot added the needs-rebase label Sep 23, 2026
lucifer1004 and others added 2 commits September 22, 2026 23:23
@lucifer1004 Kimi Code
[DSpark] Support pipeline-parallel targets in aggregated serving
a901a5d 
Draft tokens under PP: the last stage runs the speculator and now
broadcasts the fresh drafts to earlier stages (whose next verification
step would otherwise embed stale buffer contents), with the double-post
on the pp_broadcast group gated out for the speculator-less diffusion
path. Non-last stages JIT-compile the deferred post-update kernel during
warmup so its first compile cannot deadlock the pipeline mid-serving.

Draft embedding under PP: the target's embedding table lives on the
first stage, so DeepSeek-V4/Kimi-K3 DSpark drafters load their own copy
from the checkpoint (loads_own_embed_under_pp) instead of aliasing.

Padded graph batch safety: the DFlash prepare-inputs kernel now clears
input_ids/positions and sets is_padding on CUDA-graph padding rows, and
the DSv4 top-k router zeroes padded-row selections instead of reading
uninitialized state.

Co-authored-by: Kimi Code <noreply@moonshot.cn>
Signed-off-by: Zihua Wu <13583761+lucifer1004@users.noreply.github.com>
@lucifer1004 Kimi Code
Drop the draft self-embedding path in favor of vllm-project#50514
b6cc5a2 
On current main the target model retains its embedding table on the last
PP rank when speculative decoding is active (spec_decode_needs_target_embed
covers dspark), so the last-stage drafter aliases it via
maybe_share_target_embed. loads_own_embed_under_pp is redundant.

Co-authored-by: Kimi Code <noreply@moonshot.cn>
Signed-off-by: Zihua Wu <13583761+lucifer1004@users.noreply.github.com>
@mergify mergify Bot removed the needs-rebase label Sep 23, 2026
@GirasoleY GirasoleY mentioned this pull request Sep 23, 2026
Open
GirasoleY
GirasoleY approved these changes Sep 23, 2026
View reviewed changes

@GirasoleY GirasoleY left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The functionality is correct. Thanks for contributing :)

I opened a PR to change the implementation a bit so we do not need to disable/enable state during capture. Feel free to review and merge it.

@zyongye

zyongye commented Sep 23, 2026

Copy link
Copy Markdown
Member

/ci run

@zyongye
zyongye enabled auto-merge (squash) September 23, 2026 07:28
@github-actions

github-actions Bot commented Sep 23, 2026

Copy link
Copy Markdown

✅ Triggered Buildkite CI #90607 for commit 0dcd9a0f20f5.

@github-actions github-actions Bot added the ready ONLY add when PR is ready to merge/full CI is needed label Sep 23, 2026
@vllm-bot
vllm-bot merged commit 157bcb7 into vllm-project:main Sep 23, 2026
203 of 205 checks passed
@github-project-automation github-project-automation Bot moved this from Backlog to Done in Sprint - DFlash Sep 23, 2026
@njhill njhill mentioned this pull request Sep 24, 2026
Merged
njhill added a commit to njhill/vllm that referenced this pull request Sep 24, 2026
@njhill
Revert "[DSpark] Support pipeline-parallel targets in aggregated serv…
99c9ad2 
…ing (vllm-project#56956)"

This reverts commit 157bcb7.
@lucifer1004 lucifer1004 mentioned this pull request Sep 24, 2026
Open
vllm-bot pushed a commit that referenced this pull request Sep 24, 2026
@njhill
Revert "[DSpark] Support pipeline-parallel targets in aggregated serv…
09fe178 
…ing (#56956)" (#58484)
@njhill

njhill commented Sep 24, 2026

Copy link
Copy Markdown
Member

@lucifer1004 apologies we reverted this because it needs some more streamlining. We're very sensitive to adding complexity to the core model runner files. Please re-open and I can help iterate some more on it!

lucifer1004 added a commit to lucifer1004/vllm that referenced this pull request Sep 25, 2026
@lucifer1004
Reapply "[DSpark] Support pipeline-parallel targets in aggregated ser…
a4c7799 
…ving (vllm-project#56956)" (vllm-project#58484)

This reverts commit 09fe178.
lucifer1004 added a commit to lucifer1004/vllm that referenced this pull request Sep 25, 2026
@lucifer1004 @GirasoleY Kimi Code
[DSpark] Support PP targets in aggregated serving and PD prefill with…
2d3d31d 
… KV transfer

Rebuild of vllm-project#56957 as a self-contained PR: vllm-project#56956 was reverted by vllm-project#58484 for
further streamlining, so this branch now carries the full stack on current
main:

- Reapply vllm-project#56956 (DSpark IFB PP): PPHandler sampled-token/draft broadcast,
  deferred post-update warmup coverage, DSv4 router padding.
- DSpark prefill-only producer for PD: SpeculativeConfig.is_dspark_prefill_only
  + target_kv_transfer_config, NIXL guard, materialize_context_kv() called in
  place of propose() on the producer, drafter runs context_kv_only there.
- Merge with vllm-project#57632 (context K/V precompute captured in the draft CUDA graph):
  the shared prepare half of propose()/materialize_context_kv() is factored
  into _prepare_draft_inputs(); propose() keeps the graph-aware store, the
  producer stores eagerly via _precompute_context_kv().
- Fix _post_update_kernel warmup to use the int32 idx_mapping serving builds.

Co-authored-by: Summer Yang <girasoleyang@gmail.com>
Co-authored-by: Kimi Code <noreply@moonshot.cn>
Signed-off-by: Zihua Wu <13583761+lucifer1004@users.noreply.github.com>
@lucifer1004

lucifer1004 commented Sep 25, 2026

Copy link
Copy Markdown
Contributor Author

@njhill Thanks for the heads-up. Since the revert landed while #56957 (the stacked PD half) was already in review, I've rebuilt #56957 as a self-contained PR that re-applies this change and includes the PD producer path on current main (also merging cleanly with #57632's graph-captured context K/V store). Let's continue the streamlining iteration there — happy to work through the model-runner complexity concerns with you.

@bluemelov1 bluemelov1 mentioned this pull request Sep 25, 2026
Open
kristobalus added a commit to kristobalus/vllm that referenced this pull request Sep 25, 2026
@kristobalus @hlin99
@Wauplin @claude @LucasWilkinson @codex @afriedri @dllehr-amd @shen-shanshan @mergify @Sip4818 @DarkLight1337 @djramic @AndreasKaratzas @karen-sy @NickLucche @mjkvaak-amd @matteso1 @LiuYinfeng01 @vllm-agent @kimi-no-na-wa @garrett361 @taneem-ibrahim @Yuzu23 @Thangnguyenvn98 Kimi Code @robertgshaw2-redhat Robert Shaw @GirasoleY @ZJY0516 @Navjot10 @taking-lying-flat @wjabbour @khluu @njhill @chaojun-zhang @alec-flowers @BugenZhao @liusy58 @Isotr0py @wangyicong52 @aoshen02 @majunze2001 @gangula-karthik @factory-droid @jperezdealgaba @shaohuaxi @Sunt-ing @cursoragent @gcanlin @gau-nernst @jeejeelee @JaredforReal @Leoyzen @louie-tsai @hmellor @ubwzwd @arpera @zxd1997066 @jikunshang @linnea-lin-00638949 CNE Pierre FICHEPOIL @sfeng33 @kushaldabbe opencode @mgoin @Juntian777 @Yatimai @yewentao256 @stefankoncarevic @Fangzhou-Ai @guanxingithub @jiakangkangfuzhe @divakar-amd @hclsys @lucamotz @mmastrac @razorback16 @atalman Kimi @MichaelLapshin @rasmith @jhu960213 @jamesETsmith @tpopp @ywang96 @gty111 @mganczarenko @qiching @benchislett @tangzzycc @S1ro1 @ivanium @zyongye @lijipeng787 @web-flow @lucifer1004 @yma11 @nikhilkulkarni1755 @chaunceyjiang @QwertyJack @ShuoleiWang @Ankit-Jaiswal-AMD @HDCharles @jinzhen-lin @fxmarty-amd @vorapolsiloai @sahirema @okorzh-amd @Dao007forever @simondanielsson @BPbruce @LioEinaudi @divyvasal @simpleqt @aarushjain29 @blipbyte @pisceskkk @jhaotingc @karya0 @orozery @farzad-elastix @bigPYJ1151 @wzhao18 @biswapanda @tianmu-li @grYe99 grYe99 @talorabr @jackLei0901 @Yejing-Lai @Priyjain-amd @reidliu41 @RyanMa29 @ganeshr10 @KEYS-A15 @liuzijing2014 @zhenwei-intel @LinzeShi @eopXD @bohnstingl @nightcityblade nightcityblade @UNIDY2002 @vcave @shikamd123 @AARONKANG04 @CZT0 @faaany @snadampal @microslaw @haosenwang1018 @netanel-haber @khushali9 @sammaji @SIDDARTHAREDDY8 @samuelkim7 @100milliongold @gongwei-130 @jthomson04 @fululi12 @yisustc @wangxiyuan @mkunredd Mahesh Kunreddi @mrodden opencode+deepseek-v4-flash+vllm @noooop @valarLip @jryberg @shallow10 @200lz @markmc
Main (#1)
e5d7ecf 
* [Bugfix] Fix external LB DP rank handling when replicas share nodes (#53743)

Signed-off-by: Tony Lin <tony.lin@intel.com>

* [docs] Fix legacy hf CLI references (vllm) (#57958)

Signed-off-by: Wauplin <lucainp@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

* [Bugfix][NIXL] Fix DCP pulls across MLA cache regions (#57389)

Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>
Co-authored-by: OpenAI Codex <noreply@openai.com>

* [ROCm] Refactor tuned gemms (#55001)

Signed-off-by: Andy Friedrich <afriedri@amd.com>
Signed-off-by: afriedri <afriedri@amd.com>
Co-authored-by: Douglas Lehr <91553416+dllehr-amd@users.noreply.github.com>
Co-authored-by: Shanshan Shen <467638484@qq.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

* [Bugfix] unskip InternViT test for transformers v5 compatibility (#55767)

Signed-off-by: sahil <sahil@example.com>
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>

* [MM] Move get_dummy_processor_inputs into MM processor (#57967)

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

* [ROCm][CI] Use ROCm backend for DeepSeek V4.1 ViT test (#57931)

Signed-off-by: Djordje Ramic <djoramic@amd.com>
Co-authored-by: Andreas Karatzas <akaratza@amd.com>

* [Feature] Add first-class KV hints request envelope for programmatic KV management (#53423)

Signed-off-by: Karen Chung <karenc@nvidia.com>

* [Docs] Add an Engram feature page explaining Engram usage in vLLM (#57910)

Signed-off-by: NickLucche <nicolo.lucchesi@mistral.ai>

* [ROCm][Perf] Route the fused shared-expert gate GEMM through the platform dispatcher (#54185)

Signed-off-by: Mikko Tukiainen <Mikko.Tukiainen@amd.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

* [Engram] Drop redundant VLLM_PLE_CPU_OFFLOAD env var (#57937)

Signed-off-by: NickLucche <nicolo.lucchesi@mistral.ai>

* [Bugfix][MoE] Reject hash routing for unsupported monolithic backends (#57867)

Signed-off-by: Andreas Karatzas <Andreas.Karatzas@amd.com>
Co-authored-by: OpenAI Codex <noreply@openai.com>

* [Bugfix][ROCm] Reject unsupported EP for monolithic AITER MXFP4 MoE (#57866)

Signed-off-by: Andreas Karatzas <Andreas.Karatzas@amd.com>
Co-authored-by: OpenAI Codex <noreply@openai.com>

* [Docker] Use zstd for CI images and offer a Docker Hub variant (#55608)

Signed-off-by: Nils Matteson <nilsmatteson@icloud.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

* [Kimi-K3][AMD] Return KDA and MLA projection outputs directly (#50592)

Signed-off-by: Liuyinfeng01 <yinfeliu@amd.com>
Co-authored-by: Liuyinfeng01 <199041580+LiuYinfeng01@users.noreply.github.com>
Co-authored-by: Andreas Karatzas <akaratza@amd.com>

* [CI][Build] Harden triton-cpu sleef submodule fetch in CPU image build (#57871)

Signed-off-by: kimi-no-na-wa <kimi-no-na-wa@users.noreply.github.com>
Co-authored-by: kimi-no-na-wa <kimi-no-na-wa@users.noreply.github.com>

* [Bugfix][Kernel] Skip the fused silu-mul block-quant fast path when a swiglu clamp is set (#57984)

Signed-off-by: Garrett Goon <garrett@primeintellect.ai>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

* [Frontend] Add reusable TP1 initialized-engine snapshots (#51360)

Signed-off-by: Nils Matteson <nilsmatteson@icloud.com>
Signed-off-by: Taneem Ibrahim <taneem.ibrahim@gmail.com>
Signed-off-by: Andreas Karatzas <Andreas.Karatzas@amd.com>
Co-authored-by: Taneem Ibrahim <taneem.ibrahim@gmail.com>
Co-authored-by: elehayym <52448798+Yuzu23@users.noreply.github.com>
Co-authored-by: Andreas Karatzas <akaratza@amd.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

* [Pooling] MRV2 pooling shutdown model ref (#57737)

Signed-off-by: Taneem Ibrahim <taneem.ibrahim@gmail.com>

* [CI] Split (H200) LM Eval Large Models into per-model jobs (#57965)

Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
Co-authored-by: Kimi Code <noreply@moonshot.cn>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

* [Scheduler] Soften Long Prefill Tokens Threshhold (#57951)

Signed-off-by: Robert Shaw <robertgshaw2@gmail.com>
Signed-off-by: Robert Shaw <robshaw@redhat.com>
Co-authored-by: Robert Shaw <robshaw@redhat.com>

* [Perf][Attention] Reduce GLM sparse MLA preparation overhead (#57458)

Signed-off-by: Summer Yang <girasoleyang@gmail.com>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Jiangyun Zhu <riverclouds.zhu@qq.com>

* [Bugfix] Annotate MTP draft KV cache groups positionally on the hybrid grouping path (#55390)

Signed-off-by: Navjot Singh <navjot.singh@shopify.com>
Co-authored-by: Codex <noreply@openai.com>

* [Bugfix][GDN] Fix stateless first-chunk classification (#51565)

Signed-off-by: taking-lying-flat <1615405@qq.com>
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com>
Co-authored-by: zjy0516 <riverclouds.zhu@qq.com>

* [CI] Add pre-commit check that new tests are tethered to Buildkite jobs (#54867)

Signed-off-by: Turner Jabbour <doubleujabbour@gmail.com>
Signed-off-by: Turner <doubleujabbour@gmail.com>
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Co-authored-by: Kevin H. Luu <khluu000@gmail.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>

* [XPU] Fix Nemotron FP8 LM-eval config: drop CUDA-only moe_backend and wire to new Buildkite job (#49685)

Signed-off-by: Chaojun Zhang <chaojun.zhang@intel.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* [Docker] Expose bundled vllm-rs on PATH (#57606)

Signed-off-by: Alec Flowers <aflowers@nvidia.com>
Signed-off-by: Alec <35311602+alec-flowers@users.noreply.github.com>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Bugen Zhao <i@bugenzhao.com>

* [Fast Start] Cache the MTP draft model in a separate daemon group (#57312)

Signed-off-by: liusy58 <mg21330037@smail.nju.edu.cn>
Signed-off-by: Isotr0py <Isotr0py@outlook.com>
Co-authored-by: Isotr0py <Isotr0py@outlook.com>

* [Rust Frontend] Add MiMo V2.5 parser support (#57933)

Co-authored-by: Codex <noreply@openai.com>
Signed-off-by: Bugen Zhao <i@bugenzhao.com>

* [Bugfix][Spec Decode] Cap DFlash/DSpark profiling query batch (#56448)

Signed-off-by: wangyicong <wangyicong@bytedance.com>

* [RL][Sleep] Retain frozen weights across level-2 sleep (#57891)

Signed-off-by: aoshen02 <aoshen02@users.noreply.github.com>
Co-authored-by: aoshen02 <aoshen02@users.noreply.github.com>
Co-authored-by: Codex <noreply@openai.com>

* [ROCm][Bugfix] Explicitly reject FSE=1 with DPA+ETP deployment for DeepSeek-V4 (#57919)

Signed-off-by: Shanshan Shen <87969357+shen-shanshan@users.noreply.github.com>

* [Spec Decode] Enable async scheduling for DFlash (#58065)

* [Frontend][Rust] Add mm-processor benchmark for Rust frontend (#51922)

Signed-off-by: Karthik Gangula <gangula-karthik@users.noreply.github.com>
Signed-off-by: gangula-karthik <gkarthik923@gmail.com>
Signed-off-by: Isotr0py <Isotr0py@outlook.com>
Co-authored-by: Karthik Gangula <gangula-karthik@users.noreply.github.com>
Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
Co-authored-by: Isotr0py <Isotr0py@outlook.com>

* [Rust Frontend] Introduce parser-owned output grammar interfaces (#55269)

Signed-off-by: Bugen Zhao <i@bugenzhao.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>

* [Security] Reject min_tokens that exceeds the filled max_tokens default (#57731)

Signed-off-by: Juan Pérez de Algaba <jperezde@redhat.com>

* [Bugfix][Frontend] Validate mixed prompt embedding mask lengths (#57006)

Signed-off-by: 子华 <huaxi.shx@alibaba-inc.com>
Co-authored-by: Codex <noreply@openai.com>

* [Bugfix][Qwen2.5-VL] Honor video fps for temporal M-RoPE (#47736)

Signed-off-by: Ting Sun <suntcrick@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

* [Rust Frontend] Build full-output grammars from initialized reasoning parsers (#57340)

Signed-off-by: Bugen Zhao <i@bugenzhao.com>
Co-authored-by: Codex <noreply@openai.com>

* [ROCm][Perf] Avoid extra reshape kernel in Qwen GDN output norm (#47842)

Signed-off-by: Mikko Tukiainen <Mikko.Tukiainen@amd.com>
Co-authored-by: Cursor <cursoragent@cursor.com>

* [Kernel] Add opt-in load-time MXFP4 dequantization (#50814)

Signed-off-by: Liuyinfeng01 <yinfeliu@amd.com>
Co-authored-by: Shanshan Shen <467638484@qq.com>

* [Rust Frontend] Separate multimodal instrumentation from request timing (#58084)

Signed-off-by: Bugen Zhao <i@bugenzhao.com>

* [Kernel][DSV4.1] Fuse MXFP8 wo_b GEMM with sequence-parallel reduce-scatter (#57428)

Signed-off-by: Thien Tran <gau.nernst@yahoo.com.sg>
Signed-off-by: Canlin <canlinguosdu@gmail.com>
Co-authored-by: Thien Tran <gau.nernst@yahoo.com.sg>
Co-authored-by: OpenAI Codex <codex@openai.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Jee Jee Li <pandaleefree@gmail.com>

* [CI] Emit a kernel symbol map from the csrc build (opt-in, for test selection) (#58097)

Signed-off-by: khluu <khluu000@gmail.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>

* [perf] wire FA and FlashMLA for sm90 GLM5Next NoPE SparseMLA (#55385)

Signed-off-by: JaredforReal <w13431838023@gmail.com>
Co-authored-by: Kimi Code <noreply@moonshot.cn>
Co-authored-by: Leoyzen <leoyzen@gmail.com>
Co-authored-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com>

* [CPU] Add device-memory-utilization CLI alias (#56547)

Signed-off-by: louie-tsai <louie.tsai@intel.com>
Signed-off-by: Louie Tsai <louie.tsai@intel.com>
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

* [Rust Frontend] Construct model-owned vision processors through specs (#58109)

Co-authored-by: Codex <noreply@openai.com>
Signed-off-by: Bugen Zhao <i@bugenzhao.com>

* [BugFix][Core] Make the structured-output grammar poll non-blocking (#55931)

Signed-off-by: ubwzwd <ubwzwd@gmail.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Artem Perevedentsev <aperevedents@nvidia.com>

* [XPU][CI]Remove model_runner_v2 test from Intel GPU CI (#58050)

Signed-off-by: zengxian <xiangdong.zeng@intel.com>
Co-authored-by: Kunshang Ji <kunshang.ji@intel.com>

* [Bugfix][Structured Outputs] Reject empty `structural_tag` at request validation (#47450)

Signed-off-by: linnea-lin-00638949 <15521435947@163.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Artem Perevedentsev <aperevedents@nvidia.com>

* [Build] Fix CUDA 12 KV connector dependency selection (#57945)

Signed-off-by: Jee Jee Li <jeejeelee@inferact.ai>

* [Bugfix][GLM-5.3-Flash] Run the dense MLP layers on the sequence-parallel shard (#58061)

Signed-off-by: Jared Wen <w13431838023@gmail.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

* [Bugfix][Structured Output] Disallow MRV1 + PP>1 + async sched + structured output (#56250)

Signed-off-by: Artem Perevedentsev <aperevedents@nvidia.com>
Co-authored-by: CNE Pierre FICHEPOIL <pierre-1.fichepoil@gendarmerie.interieur.gouv.fr>

* [Feat][XPU] VLLM_BATCH_INVARIANT support for Dense/MoE models (#55881)

Signed-off-by: Tony Lin <tony.lin@intel.com>
Co-authored-by: Flora Feng <4florafeng@gmail.com>

* [SpecDecode] Restore residual-logits comments in _resample_kernel (#58166)

Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>

* [Bugfix] Narrow AuxOutput KV restrictions to known PD connectors (#58150)

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* [Bugfix] prioritize architecture capability before DeepGEMM availability check (#58073)

Signed-off-by: Tony Lin <tony.lin@intel.com>

* [Bugfix][Attention] Avoid NaN in the Triton softcap for large attention logits (#56579)

Signed-off-by: Kushal Dabbe <72650064+kushaldabbe@users.noreply.github.com>
Co-authored-by: opencode <noreply@opencode.ai>
Co-authored-by: Misha Goin <mgoin64@gmail.com>

* [Bugfix][Engram] Fall back when /dev/shm is absent before sharing tables (#57914)

Signed-off-by: Juntian Liu <juntianl@inferact.ai>
Signed-off-by: Juntian Liu <Juntianl777@gmail.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

* [Bugfix] hadacore_transform: respect inplace parameter to fix garbage outputs with QuIP transforms (#43462)

Signed-off-by: Gilles Turpin <turpingilles15@gmail.com>
Co-authored-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com>

* [Bugfix][ROCm] Dispatch the QuantFP8 CUDA fallback on the class (#58136)

Signed-off-by: Stefan Koncarevic <Stefan.Koncarevic@amd.com>
Co-authored-by: Andreas Karatzas <akaratza@amd.com>

* [ROCm][DSv4.1][Perf] Fuse the inverse RoPE into the sparse decode reduce (#57435)

Signed-off-by: Fangzhou Ai <fangzhou.ai@amd.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

* [Fast Start] Support data parallelism in the weight cache daemon  (#57386)

Signed-off-by: liusy58 <mg21330037@smail.nju.edu.cn>
Signed-off-by: Isotr0py <Isotr0py@outlook.com>
Co-authored-by: Isotr0py <Isotr0py@outlook.com>

* [Bugfix] batch_invariant: keep non-AllReduce collectives enabled on NCCL >= 2.31 (#58179)

Signed-off-by: Guanxin Li <38149783+guanxingithub@users.noreply.github.com>
Co-authored-by: Flora Feng <4florafeng@gmail.com>

* [MRV2] Release weight offloader on shutdown (#57834)

Signed-off-by: Taneem Ibrahim <taneem.ibrahim@gmail.com>

* [Bugfix][V1] Honor enable_jit_warmup for V2 kernel warmup (#55146)

Co-authored-by: mgoin <mgoin64@gmail.com>

* [ROCm][CI] Add GELU activation for AiterExperts in the modular-kernel coverage (#58030)

Signed-off-by: Divakar Verma <divakar.verma@amd.com>
Co-authored-by: Andreas Karatzas <akaratza@amd.com>

* [Bugfix][V1] Read ModelState max_model_len from model config (#58149)

Signed-off-by: Chenglun Hu <chenglunhu@gmail.com>
Signed-off-by: mgoin <mgoin64@gmail.com>
Co-authored-by: mgoin <mgoin64@gmail.com>

* [Bugfix][Model][Spec Decode] Defer disposable GLM MTP head (#55442)

Signed-off-by: Luca Motz <luca.motz@icloud.com>

* [Refactor] Remove dead code multiple places (#58002)

Signed-off-by: yewentao256 <zhyanwentao@126.com>

* [Core] structured generation mode for DiffusionGemma model (Jev-like) (#57250)

Signed-off-by: Matt Mastracci <matthew@mastracci.com>
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
Signed-off-by: Razorback16 <razorback16@protonmail.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Lucas Wilkinson <lwilkins@redhat.com>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Razorback16 <razorback16@protonmail.com>

* [Kernel] Remove AllSpark INT8 W8A16 GEMM backend (#58001)

Signed-off-by: mgoin <mgoin64@gmail.com>

* [CI] Build the torch-nightly image on Ubuntu 24.04 (#58204)

* [Bugfix] Backport Inductor custom-op pattern matching fix (#58189)

Signed-off-by: mgoin <mgoin64@gmail.com>

* [Core] Disable JIT warmup in eager mode (#58197)

Signed-off-by: mgoin <mgoin64@gmail.com>

* [CI] Split LM Eval TurboQuant KV Cache into per-config jobs (#57113)

Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
Co-authored-by: Kimi <noreply@moonshot.ai>

* [CI] Shard (H200 MIG 18GB) Spec Decode Draft Model across whole-directory replicas (#58193)

Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* [Perf] Remove CPU-GPU sync in heterogeneous vocabulary speculative decoding (#57396)

* [ROCm][Build][The Rock] Bump Triton version to 3.8.x tip-of-tree with source build in The Rock image (#58006)

Signed-off-by: Randall Smith <Randall.Smith@amd.com>

* [Bugfix][ROCm] Use the platform FP8 range in the concat MLA q test (#58153)

Co-authored-by: Andreas Karatzas <akaratza@amd.com>

* [ROCm][Model][Bugfix] Enable GLM-5.2-MXFP4 on the deepseek_v32 path and fix sparse attention correctness (#51915)

Signed-off-by: Jack Hu <Jack.Hu@amd.com>
Signed-off-by: Jack Hu <jack.hu@amd.com>
Signed-off-by: Douglas Lehr <Doug.Lehr@amd.com>
Co-authored-by: James E T Smith <jamesETsmith@users.noreply.github.com>
Co-authored-by: Douglas Lehr <91553416+dllehr-amd@users.noreply.github.com>
Co-authored-by: Douglas Lehr <Doug.Lehr@amd.com>

* [ROCm] Use silu_and_mul_with_clamp's torch._C op (#52052)

Signed-off-by: Tres Popp <tres.popp@amd.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Shanshan Shen <467638484@qq.com>
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Co-authored-by: Andreas Karatzas <akaratza@amd.com>

* [Bugfix] Skip VllmConfig re-validation for with_hf_config submodel views (#58212)

Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Kimi Code <noreply@moonshot.ai>
Co-authored-by: Roger Wang <rogerw@inferact.ai>

* [EPD] Support metadata-only audio inputs (#57887)

Signed-off-by: Tianyu Guo <guoty@inferact.ai>
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>

* [ROCm][DSv4][Perf] Fuse the inverse RoPE into the sparse decode reduce (#57451)

Signed-off-by: Fangzhou Ai <fangzhou.ai@amd.com>

* [ROCm][Compile] Support BF16 AsyncTP fusion (#58098)

Signed-off-by: Andreas Karatzas <akaratza@amd.com>

* [CI][Bugfix] Update IPC test caller for #57312's _apply_entries signature (#58107)

Signed-off-by: kimi-no-na-wa <kimi-no-na-wa@users.noreply.github.com>
Co-authored-by: kimi-no-na-wa <kimi-no-na-wa@users.noreply.github.com>
Co-authored-by: Kevin H. Luu <khluu000@gmail.com>

* [ROCm][CI] Stage G gating (#50922)

Signed-off-by: Andreas Karatzas <akaratza@amd.com>
Signed-off-by: Andreas Karatzas <Andreas.Karatzas@amd.com>
Co-authored-by: OpenAI Codex <codex@openai.com>

* [Bugfix] Set worker runtime threads before profiling and compilation (#55891)

Signed-off-by: Andreas Karatzas <andreas.karatzas@protonmail.com>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>

* [XPU] Wire up SYCL apply_rotary_emb kernel in ApplyRotaryEmb (#55721)

Signed-off-by: Michal Ganczarenko <michal.ganczarenko@intel.com>
Signed-off-by: Michał Ganczarenko <michal.ganczarenko@intel.com>
Co-authored-by: Claude <noreply@anthropic.com>

* [Compilation] Fix QuTLASS compilation with PyTorch 2.13 (#58173)

Signed-off-by: yewentao256 <zhyanwentao@126.com>

* [Spec decode] Support variable-length decode for Kimi-K3 adaptive ver (#52988)

Signed-off-by: Albert Cheng <albecheng@nvidia.com>
Signed-off-by: Albert Cheng (Engrg-Hardware 1) <albecheng@login-bia01.bia.clusters.nvidia.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Benjamin Chislett <bchislett@nvidia.com>

* [XPU][CI] Deselect tests/v1/spec_decode/test_mtp.py::test_glm_mtp_defers_lm_head (#58237)

Signed-off-by: zengxian <xiangdong.zeng@intel.com>

* [MoE] Use GateLinear for all MoE models (#58234)

Signed-off-by: Jee Jee Li <jeejeelee@inferact.ai>

* [Bugfix][Frontend] Keep length finish_reason for max_tokens-truncated streaming tool calls (#46303)

Signed-off-by: Ting Sun <suntcrick@gmail.com>

* [ROCm][Perf] Use wvSplitK for single-output GEMMs (#53283)

Signed-off-by: tangzzycc <3081129260@qq.com>

* [Tests] Select V2 for diffusion scheduler unit tests (#58272)

Signed-off-by: Andreas Karatzas <Andreas.Karatzas@amd.com>
Co-authored-by: OpenAI Codex <codex@openai.com>

* [Quantization] Select per-token NVFP4 MoE backends explicitly (#57176)

Signed-off-by: aoshen02 <aoshen@inferact.ai>
Signed-off-by: S1ro1 <matej.sirovatka@gmail.com>
Signed-off-by: aoshen02 <aoshen02@users.noreply.github.com>
Co-authored-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: OpenAI Codex <noreply@openai.com>
Co-authored-by: aoshen02 <aoshen02@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

* [ROCm][CI] Mirror the three TurboQuant evaluation groups on MI355 (#58282)

Signed-off-by: Andreas Karatzas <akaratza@amd.com>
Co-authored-by: OpenAI Codex <noreply@openai.com>

* [ROCm][CI] Add MI355 dense NVFP4 and MoRI kernel mirrors (#58281)

Signed-off-by: Andreas Karatzas <akaratza@amd.com>
Co-authored-by: OpenAI Codex <noreply@openai.com>

* [Mooncake] Address review nits from #56855 (#57174)

Signed-off-by: NickLucche <nicolo.lucchesi@mistral.ai>
Co-authored-by: Yifan Qiao <17067717+ivanium@users.noreply.github.com>

* [Refactor][Quantization] Make FP8 and MLA weight transforms reusable pure functions (#57732)

Signed-off-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: Yongye Zhu <zyy1102000@gmail.com>

* [ROCm][Compile] Fuse AITER static FP8 attention output (#58099)

Signed-off-by: Andreas Karatzas <akaratza@amd.com>
Co-authored-by: OpenAI Codex <noreply@openai.com>

* [ROCm][Bugfix] Register MRV2 sampler JIT warmups (#58092)

Signed-off-by: Andreas Karatzas <akaratza@amd.com>
Co-authored-by: OpenAI Codex <codex@openai.com>

* [Perf][Attention] Avoid CPU-GPU sync in DCP sequence lengths (#58169)

Signed-off-by: zjy0516 <riverclouds.zhu@qq.com>
Co-authored-by: OpenAI Codex <noreply@openai.com>

* [CI][Bugfix] Extend groupwise rms_norm scale tolerance to CUDA (#58252)

Signed-off-by: kimi-no-na-wa <kimi-no-na-wa@users.noreply.github.com>
Co-authored-by: kimi-no-na-wa <kimi-no-na-wa@users.noreply.github.com>

* [Perf][ROCm][Attention] Narrow the Triton prefill-attention KV tile on RDNA3/RDNA4 (#58225)

Signed-off-by: Jipeng Li <jipengli@amd.com>
Co-authored-by: GitHub Copilot CLI <noreply@github.com>
Co-authored-by: Andreas Karatzas <akaratza@amd.com>

* [ROCm][Bugfix] Keep zero MiniMax MXFP8 activation blocks finite (#58089)

Signed-off-by: Andreas Karatzas <akaratza@amd.com>

* [ROCm][CI] Include Python tooling in ROCm CI artifacts (#58271)

Signed-off-by: Andreas Karatzas <Andreas.Karatzas@amd.com>
Co-authored-by: OpenAI Codex <codex@openai.com>

* [MRV2] Miscellaneous code cleanup (#57980)

* [Bugfix][SM120][MLA] Support NoPE sparse MLA (GLM-5.3-Flash) on the FlashInfer SM120 backend (#55277)

Signed-off-by: Zihua Wu <13583761+lucifer1004@users.noreply.github.com>
Co-authored-by: Kimi Code <noreply@moonshot.cn>

* [XPU] upgrade to PyTorch 2.14 (#56013)

Signed-off-by: Yan Ma <yan.ma@intel.com>
Signed-off-by: Kunshang Ji <kunshang.ji@intel.com>
Co-authored-by: Kunshang Ji <kunshang.ji@intel.com>

* [ROCm][Test] Check GDN prefill numerics and output ownership (#58091)

Signed-off-by: Andreas Karatzas <akaratza@amd.com>

* [Bugfix] Disable prefix caching for encoder-only before model config hooks (#58287)

Signed-off-by: Tianyu Guo <guoty@inferact.ai>

* [Bugfix][Tool Parser] Migrate Granite to the streaming Parser Engine (#49648)

Signed-off-by: Nikhil Kulkarni <nikhilkulkarni1755@gmail.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Chauncey <chaunceyjiang@gmail.com>

* [DSpark] Support pipeline-parallel targets in aggregated serving (#56956)

Signed-off-by: Zihua Wu <13583761+lucifer1004@users.noreply.github.com>
Co-authored-by: Kimi Code <noreply@moonshot.cn>
Co-authored-by: Yongye Zhu <zyy1102000@gmail.com>

* [Feature][Frontend] Add DeepSeek-V4 FIM completion rendering (#44229)

Signed-off-by: QwertyJack <7554089+QwertyJack@users.noreply.github.com>
Co-authored-by: QwertyJack <7554089+QwertyJack@users.noreply.github.com>
Co-authored-by: Chauncey <chaunceyjiang@gmail.com>

* [Perf][MoE] Skip top-k slots routed to non-local experts in TritonExp… (#58051)

Signed-off-by: Shuolei Wang <shuoleiwang123@gmail.com>
Signed-off-by: Shuolei Wang <948904026@qq.com>

* [CPU] Adds support for fp32 attention sinks (#56252)

Signed-off-by: Ankit Jaiswal <ankit.jaiswal@amd.com>
Co-authored-by: Douglas Lehr <91553416+dllehr-amd@users.noreply.github.com>

* [Quantization] Enable humming wNaM asymmetric quant (zero_point) with compressed-tensors (#46528)

* [Quantization][Bugfix] Bump humming-kernels to 0.1.16 (#58054)

Signed-off-by: jinzhen.ljz <jinzhen.ljz@antgroup.com>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Misha Goin <mgoin64@gmail.com>

* [Quark] Remove quark-specific silent online quantization (#51800)

Signed-off-by: Felix Marty <Felix.Marty@amd.com>
Signed-off-by: mgoin <mgoin64@gmail.com>
Co-authored-by: mgoin <mgoin64@gmail.com>

* [ROCm][Perf] Extend QK-norm/RoPE/KV-cache fusion to MRoPE (#50212)

Signed-off-by: Vorapol Assavasangthong <Vorapol.Assavasangthong@amd.com>
Co-authored-by: Santosh Hiremath <Santosh.Hiremath@amd.com>
Co-authored-by: Douglas Lehr <91553416+dllehr-amd@users.noreply.github.com>

* [ROCm][CI] Validate Mooncake and NIXL prefill/decode accuracy (#58095)

Signed-off-by: Andreas Karatzas <akaratza@amd.com>
Co-authored-by: OpenAI Codex <codex@openai.com>

* [CI][ROCm] Add an MI355 Kimi-K3 unit test group (#58012)

Signed-off-by: Oxana Korzh <okorzh@amd.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Andreas Karatzas <akaratza@amd.com>

* [Bugfix][NIXL] Restore successful push completion reporting (#58188)

Signed-off-by: Dao Le <Dao007forever@gmail.com>
Co-authored-by: Codex <noreply@openai.com>

* [Bugfix][ROCm] Fix startup OOM in AITER MLA FP8 prefill workspace sizing (#57923)

Signed-off-by: simondanielsson <simon.danielsson99@hotmail.com>

* Doc: add DiffusionGemma to supported models (#46466)

Signed-off-by: Bruce <Bruce798858117@gmail.com>
Signed-off-by: Misha Goin <mgoin64@gmail.com>
Co-authored-by: Misha Goin <mgoin64@gmail.com>

* [Perf] Use breakable CUDA graphs (no torch.compile) by default under VLLM_BATCH_INVARIANT so the tuned matmul configs see the runtime M (#57586)

Signed-off-by: LioEinaudi <zhao3024667639@gmail.com>

* [CI] Select one GPU for the H200 initialized snapshot E2E step (#58351)

Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* [Bugfix] Pick a KV block size supported by every attention backend (#49845)

Signed-off-by: Divy <divy@coralbricks.ai>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

* [Docs] Fix docstring typos (output_dytpe, kwrags, Abbrivations) (#55936)

Signed-off-by: simpleqt <89645338+simpleqt@users.noreply.github.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

* [ROCm][Test] Cover MoRI graph replay and output lifetime (#58093)

Signed-off-by: Andreas Karatzas <akaratza@amd.com>

* [ROCm][Bugfix] Fix TileLang mHC fused RMSNorm on 64-wide wavefronts (#58419)

Signed-off-by: Djordje Ramic <djoramic@amd.com>
Co-authored-by: Andreas Karatzas <akaratza@amd.com>

* [CI][Bugfix] Limit MRV2 sampler JIT warmup registration to ROCm (#58465)

Signed-off-by: khluu <khluu000@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

* [CI] Disable JIT warmup by default in VllmRunner (#58452)

Signed-off-by: mgoin <mgoin64@gmail.com>

* [Bugfix][MRV2] Align dummy idx_mapping dtype to avoid runtime jit (#58462)

Signed-off-by: Nick Hill <nickhill123@gmail.com>

* [5/12][ci-selector][CI] Skip the Proton GPU test when another CUPTI tool is injected (#58455)

Signed-off-by: khluu <khluu000@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

* [CI] Share BF16 baselines across quantization comparison tests (#58469)

Signed-off-by: Aarushi Jain <Aarushi.Jain2@amd.com>
Co-authored-by: Andreas Karatzas <akaratza@amd.com>

* [Bugfix][DSA] Bound DeepSelect sentinel columns in the sparse top-k remap (#58215)

Signed-off-by: Juntian Liu <Juntianl777@gmail.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* [Test][Determinism] Cover chunked prefill in the batch-invariance suite (#55612)

Signed-off-by: Bob Ok <49168652+blipbyte@users.noreply.github.com>
Co-authored-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com>

* [CI][ROCM] Add the Fusion E2E TP2 Quick group on MI355, and the AITER MLA fix it needs (#58369)

Signed-off-by: Stefan Koncarevic <Stefan.Koncarevic@amd.com>
Signed-off-by: Andreas Karatzas <andreas.karatzas@protonmail.com>
Co-authored-by: Andreas Karatzas <akaratza@amd.com>
Co-authored-by: Andreas Karatzas <andreas.karatzas@protonmail.com>
Co-authored-by: Codex <noreply@openai.com>

* [Scheduler] Tune --long-prefill-token-threshold adaptiveness (#58459)

Signed-off-by: Robert Shaw <robertgshaw2@gmail.com>

* [PCP] Support prefill context parallelism with data parallelism (#57075)

Signed-off-by: QiuChunshuo <qiuchunshuo@huawei.com>
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
Signed-off-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com>
Co-authored-by: QiuChunshuo <qiuchunshuo@huawei.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

* [ROCm] Give turboquant boundary layers a layout-compatible backend (#54988)

Signed-off-by: Stefan Koncarevic <stefan.koncarevic@amd.com>
Signed-off-by: Andreas Karatzas <akaratza@amd.com>
Co-authored-by: Andreas Karatzas <akaratza@amd.com>
Co-authored-by: Codex <noreply@openai.com>

* [Kernel] Resubmit PR 48666 - Gemma4 FP8 KV FA4 head dim 512 backend selection (#53175)

Signed-off-by: Jhao-Ting Chen <jhaotingc@nvidia.com>

* [Bugfix][KV Offload] Retain offload event metadata through batch translation (#57453)

Signed-off-by: Kapil Arya <kapila@nvidia.com>
Signed-off-by: Kapil Arya <kapil.arya.17@gmail.com>
Co-authored-by: Or Ozeri <or@ozery.com>

* Fix full logprobs in token-in/token-out responses (#58488)

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* [Bugfix][CPU][MoE] Fix out-of-bounds write and segfault when router weights are fp32 (#56168)

Signed-off-by: Farzad Abdolhosseini <farzad@elastix.ai>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Li, Jiang <jiang1.li@intel.com>

* [Rust Frontend] Recognize new frontend-owned serve args as unsupported or no-op (#58330)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Bugen Zhao <i@bugenzhao.com>

* [Rust Frontend] Accept custom chat roles for HF templates (#58311)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Bugen Zhao <i@bugenzhao.com>

* [Dependency] Upgrade FlashInfer version to 0.7.0 (#58069)

Signed-off-by: wzhao18 <wzhao18.sz@gmail.com>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Yongye Zhu <zyy1102000@gmail.com>
Co-authored-by: Kevin H. Luu <khluu000@gmail.com>

* [Bugfix][Spec Decode] Separate DSpark width from MTP stage validation (#54631)

Signed-off-by: Luca Motz <luca.motz@icloud.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

* [Rust Frontend] Pass vision preprocessing context for Nemotron-H (#57634)

Pass the remaining engine context-length budget to model-owned vision processors through VisionPreprocessingContext. Preserve Nemotron batched engine fields and recognize llm_config as a text_config alias.

Use the merged upstream llm-multimodal revision f0985ef65967615db2c79279aa07818499301bfd.

Co-authored-by: Codex <noreply@openai.com>
Signed-off-by: Bugen Zhao <i@bugenzhao.com>

* [Rust Frontend] Support `--sse-keep-alive-interval` (#58306)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Signed-off-by: Bugen Zhao <i@bugenzhao.com>

* [Compile][CI] Honor Triton cache overrides and add AMD timeout headroom (#58474)

Signed-off-by: Andreas Karatzas <andreas.karatzas@protonmail.com>
Co-authored-by: Codex <noreply@openai.com>

* [CPU][GDN] Support NIXL DS convolution-state layout (#53300)

Signed-off-by: Li, Tianmu <tianmu.li@intel.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
Co-authored-by: Li, Jiang <jiang1.li@intel.com>

* [ROCm][CI] Add the MI355 TurboQuant t3nc mirror (#58432)

Signed-off-by: Andreas Karatzas <andreas.karatzas@protonmail.com>
Co-authored-by: OpenAI Codex <noreply@openai.com>

* [EPD][Model Loader] Skip language-model checkpoint shards for `--mm-encoder-only` (#58086)

Signed-off-by: grYe99 <guorongye99@gmail.com>
Co-authored-by: grYe99 <guorongye99@gmail.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>

* [Perf] Use Conv3dLayer for MiniMax M3 patch embedding (#58512)

Signed-off-by: OpenAI Codex <codex@openai.com>
Co-authored-by: OpenAI Codex <codex@openai.com>

* [Feature][Frontend] Request JSON body debug logging on `--enable-log-requests` flag (#58163)

Signed-off-by: talora <talora@nvidia.com>

* [CPU] Use pre-built triton (#58140)

Signed-off-by: jiang1.li <jiang1.li@intel.com>

* [Bugfix][V1] Reject encoder-cache hits with mismatched embedding counts (#57696)

Signed-off-by: jackLei0901 <42642542+jackLei0901@users.noreply.github.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

* [Bugfix] Capture prefill kernels for mixed FULL graphs (#58275)

Signed-off-by: Andreas Karatzas <Andreas.Karatzas@amd.com>
Co-authored-by: OpenAI Codex <codex@openai.com>

* [Bugfix][CI] Fix the flaky sharded-sampling tests, and the engine teardown need (#58342)

Signed-off-by: Stefan Koncarevic <Stefan.Koncarevic@amd.com>
Signed-off-by: Andreas Karatzas <Andreas.Karatzas@amd.com>
Co-authored-by: Andreas Karatzas <Andreas.Karatzas@amd.com>
Co-authored-by: Andreas Karatzas <akaratza@amd.com>

* [Bugfix][XPU] store the pointer raw bit pattern instead of its numeric value (#54514)

Signed-off-by: Lai, Yejing <yejing.lai@intel.com>
Co-authored-by: Kunshang Ji <kunshang.ji@intel.com>

* [Attention][CPU] Run Zen CPU encoder attention on zentorch SDPA (#54508)

Signed-off-by: priyansh jain <priyansh.jain2@amd.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Andreas Karatzas <akaratza@amd.com>

* [MRV2] Validate MRV2 entrypoint logits processors (#57728)

Signed-off-by: Taneem Ibrahim <taneem.ibrahim@gmail.com>

* [Bugfix][Rust Frontend] Prevent MM timing from enabling debug tracing (#58378)

Co-authored-by: Bugen Zhao <i@bugenzhao.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: reidliu41 <reid201711@gmail.com>
Signed-off-by: Bugen Zhao <i@bugenzhao.com>

* [XPU][UT] Align HF and vLLM inputs for Qwen2 embedding test by preventing Sentence Transformers from applying chat template (#58117)

Signed-off-by: RyanMa29 <ziyang.ma@intel.com>

* [CPU] Gate the AVX10.2 paths on compiler support (#58133)

Signed-off-by: R <Ganesh.R@amd.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Li, Jiang <jiang1.li@intel.com>

* [Perf][Frontend] Offload streaming derender detokenization (#57528)

Signed-off-by: Shrey Gajjar <shreygajjar007@gmail.com>

* [Multimodal] Reuse the supplied tokenizer in the MiniMax-M3 VL processor (#58460)

Signed-off-by: Zijing Liu <liuzijing2014@gmail.com>

* [XPU] enable XPU GRAPH by default (#51600)

Signed-off-by: zhenwei-intel <zhenwei.liu@intel.com>

* [ROCm][DSv4.1][Perf] Emit MXFP8 from the sparse decode reduce and run wo_a as a grouped FP8 GEMM (#58456)

Signed-off-by: fai <fangzhouai@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

* Remove `.gemini/` and `CLAUDE.md` (#58541)

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* [Bugfix][Pooling] Fix JinaVL label configuration and restore multimodal tests (#57347)

Signed-off-by: Linze-Shi <linzeshi0@gmail.com>

* [Chore] Use Transformers v5 names and drop redundant processor `use_fast` (#58550)

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* [Refactor] Remove dead or duplicate tests (#58446)

Signed-off-by: yewentao256 <zhyanwentao@126.com>

* [Perf][Attention] Bound FlashInfer prefill dequantization scratch (#57918)

Signed-off-by: Yueh-Ting Chen <yueh.ting.chen@gmail.com>

* fix(config): apply presence_penalty/frequency_penalty from override-generation-config (#50769)

Signed-off-by: Chenglun Hu <chenglunhu@gmail.com>
Signed-off-by: hclsys <chenglunhu@gmail.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

* [Bugfix] Resolve the Hub revision once per repo (#56092)

Signed-off-by: Wauplin <lucainp@gmail.com>
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

* [Frontend] Remove the slow tokenizer mode (#58545)

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Revert "[DSpark] Support pipeline-parallel targets in aggregated serving (#56956)" (#58484)

* [transformer] RMSNorm matching for alternative rsqrt (#54461)

Signed-off-by: Thomas Ortner <boh@zurich.ibm.com>
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

* [Bugfix] Count unsplit Idefics3 image patches (#48760)

Signed-off-by: nightcityblade <nightcityblade@gmail.com>
Co-authored-by: nightcityblade <nightcityblade@gmail.com>
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>

* [Bugfix] Keep JIT warmup under enforce-eager when fault tolerance is on (#58593)

Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Kimi <noreply@moonshot.cn>

* [Core] Skip JIT monitor when JIT warmup is disabled (#58590)

Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Kimi Code <noreply@moonshot.cn>

* [Fast Start] Wait for weight cache daemon readiness (#58370)

* [Bugfix][Quantization] Add Humming to the W4A8 (INT4xFP8) MoE oracle (#58427)

Signed-off-by: mgoin <mgoin64@gmail.com>

* [Refactor] Move auxiliary files out of the repository root (#58572)

Signed-off-by: mgoin <mgoin64@gmail.com>

* [Cleanup] Remove online quantization support in `fp8.py` in favor of online shorthands (#53585)

Signed-off-by: Felix Marty <Felix.Marty@amd.com>
Signed-off-by: mgoin <mgoin64@gmail.com>
Co-authored-by: mgoin <mgoin64@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

* [ROCm] Fix misrouting race-condition in multi-decode P/D disagg with mori-io (#51681)

Signed-off-by: Vincent Cave <vincent.cave@amd.com>
Signed-off-by: Shiksha Patel <shikpate@amd.com>
Co-authored-by: Shiksha Patel <shikpate@amd.com>
Co-authored-by: Douglas Lehr <91553416+dllehr-amd@users.noreply.github.com>
Co-authored-by: Andreas Karatzas <akaratza@amd.com>

* [Perf] DiffusionGemma: constrained reads over the request's logprob_token_ids (#58216)

Signed-off-by: Matt Mastracci <matthew@mastracci.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Misha Goin <mgoin64@gmail.com>

* [Bugfix] Pass quant_config to DiffusionGemma's ParallelLMHead (#48521)

Signed-off-by: Aaron Kang <aaron.h.kang@icloud.com>
Co-authored-by: Misha Goin <mgoin64@gmail.com>

* [ROCm][CI] skip the ROCm MRV1 default where MRV1 cannot serve the config (#58535)

Signed-off-by: Stefan Koncarevic <Stefan.Koncarevic@amd.com>
Co-authored-by: Andreas Karatzas <akaratza@amd.com>

* [DFlash] Capture the context K/V precompute in the draft CUDA graph (#57632)

Signed-off-by: Juntian Liu <Juntianl777@gmail.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* [Bugfix][Outlines] Fix EOS termination and unconstrained masks after rejected drafts (#58612)

* [Bugfix][KV Cache] Fix incremental multimodal block hashing (#51694)

Signed-off-by: Jellow <49915976+CZT0@users.noreply.github.com>
Signed-off-by: Jellow <dvdx@foxmail.com>

* [XPU][CI] enable prompt embeds tests on XPU (#58283)

Signed-off-by: Lin, Fanli <fanli.lin@intel.com>
Signed-off-by: Fanli Lin <fanli.lin@intel.com>
Co-authored-by: Kunshang Ji <kunshang.ji@intel.com>

* [CI] Report to CRCR after all jobs finish, gated on the build's long pole (#58628)

* [PD][PushConnector] Record last activity of remotes on the D side (#52245)

Signed-off-by: Sunita Nadampalli <nadampal@amazon.com>
Co-authored-by: Nicolò Lucchesi <nicolo.lucchesi@mistral.ai>

* [BUGFIX] fix ovis2_5 multimodal tokens (#52623)

Signed-off-by: Milosz Grunwald <milosz.grunwald@intel.com>

* [Bugfix][Core] Keep every multimodal feature in the partial-block KV event (#58288)

Signed-off-by: haosenwang1018 <haosenwang1018@users.noreply.github.com>
Co-authored-by: haosenwang1018 <haosenwang1018@users.noreply.github.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>

* [ROCm][CI] Mirror the DSv4-Flash disaggregated DP EP group on MI355 (#58558)

Signed-off-by: Stefan Koncarevic <Stefan.Koncarevic@amd.com>
Co-authored-by: Andreas Karatzas <akaratza@amd.com>

* [Bugfix][Quantization] Give LM heads standard linear metadata (#58444)

Signed-off-by: mgoin <mgoin64@gmail.com>

* [Bugfix][Mamba] Restore prompt-tail prefix-cache hits with MTP (#58368)

Signed-off-by: Netanel Haber <58652339+netanel-haber@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Benjamin Chislett <bchislett@nvidia.com>

* [Perf] Parallelize registered CUDA Triton kernel warmup at startup (#58582)

Signed-off-by: mgoin <mgoin64@gmail.com>
Co-authored-by: Codex <noreply@openai.com>

* [KV Connector] Fix DecodeBench fp8 fill values and add a startup fill mode (#58472)

Signed-off-by: Zijing Liu <liuzijing2014@gmail.com>
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai>
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

* Release prompt_embeds tensor when its InputBatch slot is freed (#57988)

Signed-off-by: khushali9 <khushali.desai9@gmail.com>

* [Bugfix][KVConnector] Finalize saves on steps without a forward (#57775)

Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai>
Co-authored-by: OpenAI Codex <noreply@openai.com>
Co-authored-by: Kimi Code <noreply@moonshot.ai>

* [Bugfix][Frontend] Count reasoning tokens for Harmony, DeepSeek-V3 and Step3 parsers (#58626)

Signed-off-by: Samyabrata Maji <116789799+sammaji@users.noreply.github.com>
Co-authored-by: Flora Feng <4florafeng@gmail.com>

* [Bugfix] GLM-5.3-Flash: launch the kpool paged MQA logits in the varlen mode its schedule was built with (#55270)

Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>

* [Bugfix] Accept EOS after grammar finish in outlines backend; reject json_object at validation (#57743)

Signed-off-by: SIDDARTHA REDDY <75976672+SIDDARTHAREDDY8@users.noreply.github.com>

* [Perf] Batch Mamba2 prefill SSM state saves, removing GPU<->CPU syncs (#49371)

Signed-off-by: samuelkim7 <samuelmwkim@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>

* [CI] Run DFlash2 NVFP4 acceptance test on B200; skip it on H200 35GB MIG (#58496)

Signed-off-by: khluu <khluu000@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Misha Goin <mgoin64@gmail.com>

* [Bugfix][MRV2] Treat padded prompt tails as spec-decode rows for hybrid models (#58434)

Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* [Kimi-K3][Perf] Dispatch GEMM for vision patch embedder (#58527)

Signed-off-by: Thien Tran <gau.nernst@yahoo.com.sg>

* [Minimax-M3][Perf] Use triton_mrope for vision tower + int64 offset fix for triton_mrope (#58526)

Signed-off-by: Thien Tran <gau.nernst@yahoo.com.sg>
Co-authored-by: Kimi <noreply@moonshot.cn>
Co-authored-by: Isotr0py <Isotr0py@outlook.com>

* [Bugfix][Quantization] Refresh online NVFP4 scales before reload post-processing (#57954)

Signed-off-by: S1ro1 <matej.sirovatka@gmail.com>
Signed-off-by: aoshen02 <aoshen02@users.noreply.github.com>
Co-authored-by: aoshen02 <aoshen02@users.noreply.github.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: aoshen02 <aoshen@inferact.ai>

* [Perf][DSv4.1] Restore the fused query RMSNorm + MXFP8 quantization path (#57679)

Signed-off-by: Juntian Liu <Juntianl777@gmail.com>

* [Bugfix][LogitsProcessor] Validate ':' separator in custom logits processor FQCN (#56020)

Signed-off-by: 100milliongold <gadian88@gmail.com>

* [ROCm][CI][AITER Coverage] Harden MoE sorting-backend/dispatch env-var test matrix (#58393)

Signed-off-by: Divakar Verma <divakar.verma@amd.com>

* [gRPC] Fix ping tolerance so long non-streaming RPCs are not dropped (#55102)

Signed-off-by: Wei Gong <wei@together.ai>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>

* [Perf][Rust Frontend] Make histogram observations lock-free (#58574)

Co-authored-by: jthomson04 <jwillthomson19@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Bugen Zhao <i@bugenzhao.com>

* [ROCm][Perf] MXFP8 GEMM on native 32x32 block scales for gfx950 (#58510)

Signed-off-by: fai <fangzhouai@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

* [Bugfix][Qwen4Exp] Keep pinned PLE prefetch ids out of the CUDA graph pool (#58489)

Signed-off-by: Juntian Liu <Juntianl777@gmail.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

* [Perf][Engram] Serialize offloaded lookups and pack host tables into huge pages (#56926)

Signed-off-by: Juntian Liu <Juntianl777@gmail.com>
Co-authored-by: Kimi Code <noreply@moonshot.cn>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

* [Bugfix][Quantization] Fix MXFP8 startup crash on layers below mm_mxfp8 shape limits (#54223)

Signed-off-by: samuelkim7 <samuelmwkim@gmail.com>
Signed-off-by: mgoin <mgoin64@gmail.com>
Co-authored-by: mgoin <mgoin64@gmail.com>

* [ROCm] Cut 69 wasted contiguous copies per decode step from the skinny GEMM path (#58566)

Signed-off-by: lifulu <fululi12@amd.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>

* [ROCm][Build] Filter crate tags from vLLM version detection (#57744)

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Signed-off-by: reidliu41 <reid201711@gmail.com>

* [Perf][Distributed] Add low-SM multimem reduce-scatter for SM100/SM103 (#55072)

Signed-off-by: Yongye Zhu <zyy1102000@gmail.com>
Signed-off-by: Summer Yang <girasoleyang@gmail.com>
Co-authored-by: Summer Yang <girasoleyang@gmail.com>

* [PP][XPU]Add the flag to control microbatch feature on MRV2+PP (#55145)

Signed-off-by: yisheng <yi.sheng@intel.com>
Co-authored-by: Kunshang Ji <kunshang.ji@intel.com>

* [Feature] Triton kernel dispatcher (#43048)

Signed-off-by: wangxiyuan <wangxiyuan1007@gmail.com>

* [ROCm] Fix CI runtime and tests for MI355 DPX (#58244)

Signed-off-by: Andreas Karatzas <Andreas.Karatzas@amd.com>
Co-authored-by: Mahesh Kunreddi <mahesh.kunreddi@amd.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Andreas Karatzas <Andreas.Karatzas@amd.com>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Andreas Karatzas <akaratza@amd.com>

* [CI][ROCm] Prevent Model Executor apt stalls (#58607)

Signed-off-by: Andreas Karatzas <akaratza@amd.com>
Co-authored-by: Codex <noreply@openai.com>

* [Qwen4Exp][ROCm] PLE n-gram table CPU offload (#57497)

Signed-off-by: Mathew Odden <modden@redhat.com>
Signed-off-by: Andreas Karatzas <akaratza@amd.com>
Co-authored-by: opencode+deepseek-v4-flash+vllm <opencode+deepseek-v4-flash+vllm@example.com>
Co-authored-by: Andreas Karatzas <akaratza@amd.com>
Co-authored-by: OpenAI Codex <noreply@openai.com>

* [MM] Add Triton kernel for mm_input_normal. (#56798)

Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io>
Signed-off-by: wang.yuqi <noooop@126.com>
Signed-off-by: Isotr0py <Isotr0py@outlook.com>
Co-authored-by: Isotr0py <2037008807@qq.com>
Co-authored-by: Isotr0py <Isotr0py@outlook.com>

* [Structured Outputs] Parse Lark grammars natively in the xgrammar backend (#58321)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Signed-off-by: Bugen Zhao <i@bugenzhao.com>

* [ROCm] Credit ROCm/aiter for the block32 GEMM's packed kernel and in-launch split-K (#58659)

Signed-off-by: Lingpeng Jin <103567126+valarLip@users.noreply.github.com>
Co-authored-by: Andreas Karatzas <akaratza@amd.com>

* [Frontend] Handle Disable Thinking in /v1/messages (#58613)

Signed-off-by: jryberg <johan.ryberg@security.ntt>
Signed-off-by: Robert Shaw <robshaw@redhat.com>
Co-authored-by: jryberg <johan.ryberg@security.ntt>
Co-authored-by: Robert Shaw <robshaw@redhat.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* [Bugfix] Stop leaking the internal field name in the max_tokens validation error (#58336)

Signed-off-by: shallow10 <495593563@qq.com>

* [Bugfix][KV Cache][MLA] Align packed block strides for V3.2 sparse MLA (#55528)

Signed-off-by: lz <145014769+200lz@users.noreply.github.com>
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
Co-authored-by: Lucas Wilkinson <lwilkins@redhat.com>
Co-authored-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

* [Perf][DSv4] Fuse inverse RoPE + FP8 quant into FlashInfer sparse MLA (#58621)

Signed-off-by: Yongye Zhu <zyy1102000@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

* [UX][Frontend] Introduce `vllm preload` cli for fast restart (#56680)

Signed-off-by: Isotr0py <Isotr0py@outlook.com>

* [MoE] Defer the TRTLLM-Gen top-k finalize on the modular path (#58635)

Signed-off-by: Yongye Zhu <zyy1102000@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

* [Docs] Add return annotation to `fused_mm_input_norm_triton` (#58687)

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

* [CI] Shard (H100) Helion Kernels five ways (#58645)

Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

* [Core] Model console logging as CLI configuration (#57205)

Add `--logging-config` CLI argument which can be supplied as
JSON or using dotted arguments. The `--log-level` argument
is provided for convenience, and `--log-config-file` is deprecated
in favor of `--logging-config.pylogging_config_file`.

Signed-off-by: Mark McLoughlin <markmc@redhat.com>
Co-authored-by: AI Assistant <noreply@openai.com>

* [ROCm][CI] Pass weight_shape in MXFP8 block32 linear tests (#58698)

Signed-off-by: Djordje Ramic <djoramic@amd.com>
Co-authored-by: Andreas Karatzas <akaratza@amd.com>

---------

Signed-off-by: Tony Lin <tony.lin@intel.com>
Signed-off-by: Wauplin <lucainp@gmail.com>
Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>
Signed-off-by: Andy Friedrich <afriedri@amd.com>
Signed-off-by: afriedri <afriedri@amd.com>
Signed-off-by: sahil <sahil@example.com>
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Signed-off-by: Djordje Ramic <djoramic@amd.com>
Signed-off-by: Karen Chung <karenc@nvidia.com>
Signed-off-by: NickLucche <nicolo.lucchesi@mistral.ai>
Signed-off-by: Mikko Tukiainen <Mikko.Tukiainen@amd.com>
Signed-off-by: Andreas Karatzas <Andreas.Karatzas@amd.com>
Signed-off-by: Nils Matteson <nilsmatteson@icloud.com>
Signed-off-by: Liuyinfeng01 <yinfeliu@amd.com>
Signed-off-by: kimi-no-na-wa <kimi-no-na-wa@users.noreply.github.com>
Signed-off-by: Garrett Goon <garrett@primeintellect.ai>
Signed-off-by: Taneem Ibrahim <taneem.ibrahim@gmail.com>
Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
Signed-off-by: Robert Shaw <robertgshaw2@gmail.com>
Signed-off-by: Robert Shaw <robshaw@redhat.com>
Signed-off-by: Summer Yang <girasoleyang@gmail.com>
Signed-off-by: Navjot Singh <navjot.singh@shopify.com>
Signed-off-by: taking-lying-flat <1615405@qq.com>
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com>
Signed-off-by: Turner Jabbour <doubleujabbour@gmail.com>
Signed-off-by: Turner <doubleujabbour@gmail.com>
Signed-off-by: Chaojun Zhang <chaojun.zhang@intel.com>
Signed-off-by: Alec Flowers <aflowers@nvidia.com>
Signed-off-by: Alec <35311602+alec-flowers@users.noreply.github.com>
Signed-off-by: liusy58 <mg21330037@smail.nju.edu.cn>
Signed-off-by: Isotr0py <Isotr0py@outlook.com>
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
Signed-off-by: wangyicong <wangyicong@bytedance.com>
Signed-off-by: aoshen02 <aoshen02@users.noreply.github.com>
Signed-off-by: Shanshan Shen <87969357+shen-shanshan@users.noreply.github.com>
Signed-off-by: Karthik Gangula <gangula-karthik@users.noreply.github.com>
Signed-off-by: gangula-karthik <gkarthik923@gmail.com>
Signed-off-by: Juan Pérez de Algaba <jperezde@redhat.com>
Signed-off-by: 子华 <huaxi.shx@alibaba-inc.com>
Signed-off-by: Ting Sun <suntcrick@gmail.com>
Signed-off-by: Thien Tran <gau.nernst@yahoo.com.sg>
Signed-off-by: Canlin <canlinguosdu@gmail.com>
Signed-off-by: khluu <khluu000@gmail.com>
Signed-off-by: JaredforReal <w13431838023@gmail.com>
Signed-off-by: louie-tsai <louie.tsai@intel.com>
Signed-off-by: Louie Tsai <louie.tsai@intel.com>
Signed-off-by: ubwzwd <ubwzwd@gmail.com>
Signed-off-by: zengxian <xiangdong.zeng@intel.com>
Signed-off-by: linnea-lin-00638949 <15521435947@163.com>
Signed-off-by: Jee Jee Li <jeejeelee@inferact.ai>
Signed-off-by: Jared Wen <w13431838023@gmail.com>
Signed-off-by: Artem Perevedentsev <aperevedents@nvidia.com>
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Signed-off-by: Kushal Dabbe <72650064+kushaldabbe@users.noreply.github.com>
Signed-off-by: Juntian Liu <juntianl@inferact.ai>
Signed-off-by: Juntian Liu <Juntianl777@gmail.com>
Signed-off-by: Gilles Turpin <turpingilles15@gmail.com>
Signed-off-by: Stefan Koncarevic <Stefan.Koncarevic@amd.com>
Signed-off-by: Fangzhou Ai <fangzhou.ai@amd.com>
Signed-off-by: Guanxin Li <38149783+guanxingithub@users.noreply.github.com>
Signed-off-by: Divakar Verma <divakar.verma@amd.com>
Signed-off-by: Chenglun Hu <chenglunhu@gmail.com>
Signed-off-by: mgoin <mgoin64@gmail.com>
Signed-off-by: Luca Motz <luca.motz@icloud.com>
Signed-off-by: yewentao256 <zhyanwentao@126.com>
Signed-off-by: Matt Mastracci <matthew@mastracci.com>
Signed-off-by: Razorback16 <razorback16@protonmail.com>
Signed-off-by: Randall Smith <Randall.Smith@amd.com>
Signed-off-by: Jack Hu <Jack.Hu@amd.com>
Signed-off-by: Jack Hu <jack.hu@amd.com>
Signed-off-by: Douglas Lehr <Doug.Lehr@amd.com>
Signed-off-by: Tres Popp <tres.popp@amd.com>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Signed-off-by: Tianyu Guo <guoty@inferact.ai>
Signed-off-by: Andreas Karatzas <akaratza@amd.com>
Signed-off-by: Andreas Karatzas <andreas.karatzas@protonmail.com>
Signed-off-by: Michal Ganczarenko <michal.ganczarenko@intel.com>
Signed-off-by: Michał Ganczarenko <michal.ganczarenko@intel.com>
Signed-off-by: Albert Cheng <albecheng@nvidia.com>
Signed-off-by: Albert Cheng (Engrg-Hardware 1) <albecheng@login-bia01.bia.clusters.nvidia.com>
Signed-off-by: tangzzycc <3081129260@qq.com>
Signed-off-by: S1ro1 <matej.sirovatka@gmail.com>
Signed-off-by: Jipeng Li <jipengli@amd.com>
Signed-off-by: Zihua Wu <13583761+lucifer1004@users.noreply.github.com>
Signed-off-by: Yan Ma <yan.ma@intel.com>
Signed-off-by: Kunshang Ji <kunshang.ji@intel.com>
Signed-off-by: Nikhil Kulkarni <nikhilkulkarni1755@gmail.com>
Signed-off-by: QwertyJack <7554089+QwertyJack@users.noreply.github.com>
Signed-off-by: Shuolei Wang <shuoleiwang123@gmail.com>
Signed-off-by: Shuolei Wang <948904026@qq.com>
Signed-off-by: Ankit Jaiswal <ankit.jaiswal@amd.com>
Signed-off-by: jinzhen.ljz <jinzhen.ljz@antgroup.com>
Signed-off-by: Felix Marty <Felix.Marty@amd.com>
Signed-off-by: Vorapol Assavasangthong <Vorapol.Assavasangthong@amd.com>
Signed-off-by: Oxana Korzh <okorzh@amd.com>
Signed-off-by: Dao Le <Dao007forever@gmail.com>
Signed-off-by: simondanielsson <simon.danielsson99@hotmail.com>
Signed-off-by: Bruce <Bruce798858117@gmail.com>
Signed-off-by: Misha Goin <mgoin64@gmail.com>
Signed-off-by: LioEinaudi <zhao3024667639@gmail.com>
Signed-off-by: Divy <divy@coralbricks.ai>
Signed-off-by: simpleqt <89645338+simpleqt@users.noreply.github.com>
Signed-off-by: Aarushi Jain <Aarushi.Jain2@amd.com>
Signed-off-by: Bob Ok <49168652+blipbyte@users.noreply.github.com>
Signed-off-by: QiuChunshuo <qiuchunshuo@huawei.com>
Signed-off-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com>
Signed-off-by: Stefan Koncarevic <stefan.koncarevic@amd.com>
Signed-off-by: Jhao-Ting Chen <jhaotingc@nvidia.com>
Signed-off-by: Kapil Arya <kapila@nvidia.com>
Signed-off-by: Kapil Arya <kapil.arya.17@gmail.com>
Signed-off-by: Farzad Abdolhosseini <farzad@elastix.ai>
Signed-off-by: wzhao18 <wzhao18.sz@gmail.com>
Signed-off-by: Li, Tianmu <tianmu.li@intel.com>
Signed-off-by: grYe99 <guorongye99@gmail.com>
Signed-off-by: OpenAI Codex <codex@openai.com>
Signed-off-by: talora <talora@nvidia.com>
Signed-off-by: jiang1.li <jiang1.li@intel.com>
Signed-off-by: jackLei0901 <42642542+jackLei0901@users.noreply.github.com>
Signed-off-by: Lai, Yejing <yejing.lai@intel.com>
Signed-off-by: priyansh jain <priyansh.jain2@amd.com>
Signed-off-by: reidliu41 <reid201711@gmail.com>
Signed-off-by: RyanMa29 <ziyang.ma@intel.com>
Signed-off-by: R <Ganesh.R@amd.com>
Signed-off-by: Shrey Gajjar <shreygajjar007@gmail.com>
Signed-off-by: Zijing Liu <liuzijing2014@gmail.com>
Signed-off-by: zhenwei-intel <zhenwei.liu@intel.com>
Signed-off-by: fai <fangzhouai@gmail.com>
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Signed-off-by: Linze-Shi <linzeshi0@gmail.com>
Signed-off-by: Yueh-Ting Chen <yueh.ting.chen@gmail.com>
Signed-off-by: hclsys <chenglunhu@gmail.com>
Signed-off-by: Thomas Ortner <boh@zurich.ibm.com>
Signed-off-by: nightcityblade <nightcityblade@gmail.com>
Signed-off-by: Vincent Cave <vincent.cave@amd.com>
Signed-off-by: Shiksha Patel <shikpate@amd.com>
Signed-off-by: Aaron Kang <aaron.h.kang@icloud.com>
Signed-off-by: Jellow <49915976+CZT0@users.noreply.github.com>
Signed-off-by: Jellow <dvdx@foxmail.com>
Signed-off-by: Lin, Fanli <fanli.lin@intel.com>
Signed-off-by: Fanli Lin <fanli.lin@intel.com>
Signed-off-by: Sunita Nadampalli <nadampal@amazon.com>
Signed-off-by: Milosz Grunwald <milosz.grunwald@intel.com>
Signed-off-by: haosenwang1018 <haosenwang1018@users.noreply.github.com>
Signed-off-by: Netanel Haber <58652339+netanel-haber@users.noreply.github.com>
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai>
Signed-off-by: khushali9 <khushali.desai9@gmail.com>
Signed-off-by: Samyabrata Maji <116789799+sammaji@users.noreply.github.com>
Signed-off-by: SIDDARTHA REDDY <75976672+SIDDARTHAREDDY8@users.noreply.github.com>
Signed-off-by: samuelkim7 <samuelmwkim@gmail.com>
Signed-off-by: 100milliongold <gadian88@gmail.com>
Signed-off-by: Wei Gong <wei@together.ai>
Signed-off-by: lifulu <fululi12@amd.com>
Signed-off-by: Yongye Zhu <zyy1102000@gmail.com>
Signed-off-by: yisheng <yi.sheng@intel.com>
Signed-off-by: wangxiyuan <wangxiyuan1007@gmail.com>
Signed-off-by: Mathew Odden <modden@redhat.com>
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io>
Signed-off-by: wang.yuqi <noooop@126.com>
Signed-off-by: Lingpeng Jin <103567126+valarLip@users.noreply.github.com>
Signed-off-by: jryberg <johan.ryberg@security.ntt>
Signed-off-by: shallow10 <495593563@qq.com>
Signed-off-by: lz <145014769+200lz@users.noreply.github.com>
Signed-off-by: Mark McLoughlin <markmc@redhat.com>
Co-authored-by: Tony Lin <tony.lin@intel.com>
Co-authored-by: Lucain <lucainp@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com>
Co-authored-by: OpenAI Codex <noreply@openai.com>
Co-authored-by: afriedri <afriedri@amd.com>
Co-authored-by: Douglas Lehr <91553416+dllehr-amd@users.noreply.github.com>
Co-authored-by: Shanshan Shen <467638484@qq.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Sahil Patel <91423311+Sip4818@users.noreply.github.com>
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>
Co-authored-by: djramic <djoramic@amd.com>
Co-authored-by: Andreas Karatzas <akaratza@amd.com>
Co-authored-by: Karen Chung <karenc@nvidia.com>
Co-authored-by: Nicolò Lucchesi <nicolo.lucchesi@mistral.ai>
Co-authored-by: Mikko Tukiainen <mikko.tukiainen@amd.com>
Co-authored-by: Nils Matteson <nilsmatteson@icloud.com>
Co-authored-by: yinfengLiu <yinfeliu@amd.com>
Co-authored-by: Liuyinfeng01 <199041580+LiuYinfeng01@users.noreply.github.com>
Co-authored-by: vllm-agent <claw@inferact.ai>
Co-authored-by: kimi-no-na-wa <kimi-no-na-wa@users.noreply.github.com>
Co-authored-by: Garrett Goon <44747910+garrett361@users.noreply.github.com>
Co-authored-by: Taneem Ibrahim <taneem.ibrahim@gmail.com>
Co-authored-by: elehayym <52448798+Yuzu23@users.noreply.github.com>
Co-authored-by: Thang Nguyen <69278249+Thangnguyenvn98@users.noreply.github.com>
Co-authored-by: Kimi Code <noreply@moonshot.cn>
Co-authored-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com>
Co-authored-by: Robert Shaw <robshaw@redhat.com>
Co-authored-by: Summer Yang <girasoleyang@gmail.com>
Co-authored-by: Jiangyun Zhu <riverclouds.zhu@qq.com>
Co-authored-by: Navjot Singh <navjot.singh@uwaterloo.ca>
Co-authored-by: cherry77-cloud <1615405@qq.com>
Co-authored-by: Turner Jabbour <doubleujabbour@gmail.com>
Co-authored-by: Kevin H. Luu <khluu000@gmail.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Chaojun Zhang <chaojun.zhang@intel.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Alec <35311602+alec-flowers@users.noreply.github.com>
Co-authored-by: Bugen Zhao <i@bugenzhao.com>
Co-authored-by: siyu <mg21330037@smail.nju.edu.cn>
Co-authored-by: Isotr0py <Isotr0py@outlook.com>
Co-authored-by: Yicong Wang <wangyicong@bytedance.com>
Co-authored-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: aoshen02 <aoshen02@users.noreply.github.com>
Co-authored-by: Jeff (Junze) Ma <93145857+majunze2001@users.noreply.github.com>
Co-authored-by: karthik <56480632+gangula-karthik@users.noreply.github.com>
Co-authored-by: Karthik Gangula <gangula-karthik@users.noreply.github.com>
Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
Co-authored-by: Juan Pérez de Algaba <124347725+jperezdealgaba@users.noreply.github.com>
Co-authored-by: shaohuaxi <huaxi.shx@alibaba-inc.com>
Co-authored-by: Ting SUN <suntcrick@gmail.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Canlin Guo <canlinguosdu@gmail.com>
Co-authored-by: Thien Tran <gau.nernst@yahoo.com.sg>
Co-authored-by: OpenAI Codex <codex@openai.com>
Co-authored-by: Jee Jee Li <pandaleefree@gmail.com>
Co-authored-by: Jared Wen <w13431838023@gmail.com>
Co-authored-by: Leoyzen <leoyzen@gmail.com>
Co-authored-by: Louie Tsai <louie.tsai@intel.com>
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: ubwzwd <ubwzwd@gmail.com>
Co-authored-by: Artem Perevedentsev <aperevedents@nvidia.com>
Co-authored-by: xiangdong <40376367+zxd1997066@users.noreply.github.com>
Co-authored-by: Kunshang Ji <kunshang.ji@intel.com>
Co-authored-by: linyafeng <15521435947@163.com>
Co-authored-by: CNE Pierre FICHEPOIL <pierre-1.fichepoil@gendarmerie.interieur.gouv.fr>
Co-authored-by: Flora Feng <4florafeng@gmail.com>
Co-authored-by: Kushal <72650064+kushaldabbe@users.noreply.github.com>
Co-authored-by: opencode <noreply@opencode.ai>
Co-authored-by: Misha Goin <mgoin64@gmail.com>
Co-authored-by: Juntian Liu <Juntianl777@gmail.com>
Co-authored-by: Gilles Turpin <turpingilles15@gmail.com>
Co-authored-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com>
Co-authored-by: stefankoncarevic <Stefan.Koncarevic@amd.com>
Co-authored-by: Fangzhou Ai <31551580+Fangzhou-Ai@users.noreply.github.com>
Co-authored-by: Guanxin Li <38149783+guanxingithub@users.noreply.github.com>
Co-authored-by: pengyihang <1017861497@qq.com>
Co-authored-by: Divakar Verma <137818590+divakar-amd@users.noreply.github.com>
Co-authored-by: hcl <chenglunhu@gmail.com>
Co-authored-by: lucamotz <luca.motz@icloud.com>
Co-authored-by: Matt Mastracci <matthew@mastracci.com>
Co-authored-by: Lucas Wilkinson <lwilkins@redhat.com>
Co-authored-by: Razorback16 <razorback16@protonmail.com>
Co-authored-by: Andrey Talman <atalman@fb.com>
Co-authored-by: Kimi <noreply@moonshot.ai>
Co-authored-by: Michael Lapshin <55516685+MichaelLapshin@users.noreply.github.com>
Co-authored-by: rasmith <Randall.Smith@amd.com>
Co-authored-by: Jack Hu <jack.hu@amd.com>
Co-authored-by: James E T Smith <jamesETsmith@users.noreply.github.com>
Co-authored-by: Douglas Lehr <Doug.Lehr@amd.com>
Co-authored-by: Tres <tpopp@users.noreply.github.com>
Co-authored-by: Roger Wang <rogerw@inferact.ai>
Co-authored-by: Tianyu Guo <guoty@inferact.ai>
Co-authored-by: Michał Ganczarenko <michal.gancz…
lucifer1004 added a commit to lucifer1004/vllm that referenced this pull request Sep 25, 2026
@lucifer1004
Reapply "[DSpark] Support pipeline-parallel targets in aggregated ser…
b8c1314 
…ving (vllm-project#56956)" (vllm-project#58484)

This reverts commit 09fe178.

Signed-off-by: Zihua Wu <13583761+lucifer1004@users.noreply.github.com>
lucifer1004 added a commit to lucifer1004/vllm that referenced this pull request Sep 25, 2026
@lucifer1004 @GirasoleY Kimi Code
[DSpark] Support PP targets in aggregated serving and PD prefill with…
fac02e1 
… KV transfer

Rebuild of vllm-project#56957 as a self-contained PR: vllm-project#56956 was reverted by vllm-project#58484 for
further streamlining, so this branch now carries the full stack on current
main:

- Reapply vllm-project#56956 (DSpark IFB PP): PPHandler sampled-token/draft broadcast,
  deferred post-update warmup coverage, DSv4 router padding.
- DSpark prefill-only producer for PD: SpeculativeConfig.is_dspark_prefill_only
  + target_kv_transfer_config, NIXL guard, materialize_context_kv() called in
  place of propose() on the producer, drafter runs context_kv_only there.
- Merge with vllm-project#57632 (context K/V precompute captured in the draft CUDA graph):
  the shared prepare half of propose()/materialize_context_kv() is factored
  into _prepare_draft_inputs(); propose() keeps the graph-aware store, the
  producer stores eagerly via _precompute_context_kv().
- Fix _post_update_kernel warmup to use the int32 idx_mapping serving builds.

Co-authored-by: Summer Yang <girasoleyang@gmail.com>
Co-authored-by: Kimi Code <noreply@moonshot.cn>
Signed-off-by: Zihua Wu <13583761+lucifer1004@users.noreply.github.com>
lucifer1004 added a commit to lucifer1004/vllm that referenced this pull request Sep 25, 2026
@lucifer1004
Reapply "[DSpark] Support pipeline-parallel targets in aggregated ser…
dfd4b77 
…ving (vllm-project#56956)" (vllm-project#58484)

This reverts commit 09fe178.

Signed-off-by: Zihua Wu <13583761+lucifer1004@users.noreply.github.com>
lucifer1004 added a commit to lucifer1004/vllm that referenced this pull request Sep 25, 2026
@lucifer1004 @GirasoleY Kimi Code
[DSpark] Support PP targets in aggregated serving and PD prefill with…
7d4a12b 
… KV transfer

Rebuild of vllm-project#56957 as a self-contained PR: vllm-project#56956 was reverted by vllm-project#58484 for
further streamlining, so this branch now carries the full stack on current
main:

- Reapply vllm-project#56956 (DSpark IFB PP): PPHandler sampled-token/draft broadcast,
  deferred post-update warmup coverage, DSv4 router padding.
- DSpark prefill-only producer for PD: SpeculativeConfig.is_dspark_prefill_only
  + target_kv_transfer_config, NIXL guard, materialize_context_kv() called in
  place of propose() on the producer, drafter runs context_kv_only there.
- Merge with vllm-project#57632 (context K/V precompute captured in the draft CUDA graph):
  the shared prepare half of propose()/materialize_context_kv() is factored
  into _prepare_draft_inputs(); propose() keeps the graph-aware store, the
  producer stores eagerly via _precompute_context_kv().
- Fix _post_update_kernel warmup to use the int32 idx_mapping serving builds.

Co-authored-by: Summer Yang <girasoleyang@gmail.com>
Co-authored-by: Kimi Code <noreply@moonshot.cn>
Signed-off-by: Zihua Wu <13583761+lucifer1004@users.noreply.github.com>
lucifer1004 added a commit to lucifer1004/vllm that referenced this pull request Sep 25, 2026
@lucifer1004
Reapply "[DSpark] Support pipeline-parallel targets in aggregated ser…
b32d0e4 
…ving (vllm-project#56956)" (vllm-project#58484)

This reverts commit 09fe178.

Signed-off-by: Zihua Wu <13583761+lucifer1004@users.noreply.github.com>
lucifer1004 added a commit to lucifer1004/vllm that referenced this pull request Sep 25, 2026
@lucifer1004 @GirasoleY Kimi Code
[DSpark] Support PP targets in aggregated serving and PD prefill with…
0af416c 
… KV transfer

Rebuild of vllm-project#56957 as a self-contained PR: vllm-project#56956 was reverted by vllm-project#58484 for
further streamlining, so this branch now carries the full stack on current
main:

- Reapply vllm-project#56956 (DSpark IFB PP): PPHandler sampled-token/draft broadcast,
  deferred post-update warmup coverage, DSv4 router padding.
- DSpark prefill-only producer for PD: SpeculativeConfig.is_dspark_prefill_only
  + target_kv_transfer_config, NIXL guard, materialize_context_kv() called in
  place of propose() on the producer, drafter runs context_kv_only there.
- Merge with vllm-project#57632 (context K/V precompute captured in the draft CUDA graph):
  the shared prepare half of propose()/materialize_context_kv() is factored
  into _prepare_draft_inputs(); propose() keeps the graph-aware store, the
  producer stores eagerly via _precompute_context_kv().
- Fix _post_update_kernel warmup to use the int32 idx_mapping serving builds.

Co-authored-by: Summer Yang <girasoleyang@gmail.com>
Co-authored-by: Kimi Code <noreply@moonshot.cn>
Signed-off-by: Zihua Wu <13583761+lucifer1004@users.noreply.github.com>
lucifer1004 added a commit to lucifer1004/vllm that referenced this pull request Sep 26, 2026
@lucifer1004
Reapply "[DSpark] Support pipeline-parallel targets in aggregated ser…
00b3b52 
…ving (vllm-project#56956)" (vllm-project#58484)

This reverts commit 09fe178.

Signed-off-by: Zihua Wu <13583761+lucifer1004@users.noreply.github.com>
lucifer1004 added a commit to lucifer1004/vllm that referenced this pull request Sep 26, 2026
@lucifer1004 @GirasoleY Kimi Code
[DSpark] Support PP targets in aggregated serving and PD prefill with…
99650fa 
… KV transfer

Rebuild of vllm-project#56957 as a self-contained PR: vllm-project#56956 was reverted by vllm-project#58484 for
further streamlining, so this branch now carries the full stack on current
main:

- Reapply vllm-project#56956 (DSpark IFB PP): PPHandler sampled-token/draft broadcast,
  deferred post-update warmup coverage, DSv4 router padding.
- DSpark prefill-only producer for PD: SpeculativeConfig.is_dspark_prefill_only
  + target_kv_transfer_config, NIXL guard, materialize_context_kv() called in
  place of propose() on the producer, drafter runs context_kv_only there.
- Merge with vllm-project#57632 (context K/V precompute captured in the draft CUDA graph):
  the shared prepare half of propose()/materialize_context_kv() is factored
  into _prepare_draft_inputs(); propose() keeps the graph-aware store, the
  producer stores eagerly via _precompute_context_kv().
- Fix _post_update_kernel warmup to use the int32 idx_mapping serving builds.

Co-authored-by: Summer Yang <girasoleyang@gmail.com>
Co-authored-by: Kimi Code <noreply@moonshot.cn>
Signed-off-by: Zihua Wu <13583761+lucifer1004@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@claude claude[bot] claude[bot] left review comments
@zyongye zyongye zyongye approved these changes
@DarkLight1337 DarkLight1337 Awaiting requested review from DarkLight1337
@ywang96 ywang96 Awaiting requested review from ywang96
@AndreasKaratzas AndreasKaratzas Awaiting requested review from AndreasKaratzas AndreasKaratzas is a code owner
@mgoin mgoin Awaiting requested review from mgoin mgoin is a code owner
@tlrmchlsmth tlrmchlsmth Awaiting requested review from tlrmchlsmth tlrmchlsmth is a code owner
@WoosukKwon WoosukKwon Awaiting requested review from WoosukKwon WoosukKwon is a code owner
@yewentao256 yewentao256 Awaiting requested review from yewentao256 yewentao256 is a code owner
@njhill njhill Awaiting requested review from njhill njhill is a code owner
@pavanimajety pavanimajety Awaiting requested review from pavanimajety pavanimajety is a code owner
+1 more reviewer
@GirasoleY GirasoleY GirasoleY approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

deepseek Related to DeepSeek models dflash DSv4 k3 kimi mrv2 Model Runner V2 specific ready ONLY add when PR is ready to merge/full CI is needed speculative-decoding

Projects

Sprint - DFlash
Status: Done

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@lucifer1004 @zyongye @njhill @GirasoleY @vllm-bot